ShowTable of Contents
This section lists the currently configured limitations of the Expeditor integrator.
Message Size
The message size is currently limited to 10 MByte. This can be configured in the XPDinteg.xml file, but should only be done with care. Many large messages can dramatically slow down the processing speed.
If the Lotus Expeditor micro broker bridge does not pick up messages carrying large files from the MQ back-end messaging server, the IBM MQ Server settings at the back-end are to be evaluated. The follwing properties should be adjusted in order to be able to transfer large files:
- Increase the Max. Message Length of the
- Queue Manager under menu Properties | Extended | Maximum Message Length
- ReqOutQx at the back-end under menu Properties | Extended | Maximum Message Length
- MQ Channel under menu Properties | Extended | Maximum Message Length
- Increase the Number of Log Files used by the MQ Queue Manager under menu Properties | Log | Log primary files / Log secondary files
Message Load
The Message Load (in most cases the carried file) is also limited to 10 MByte at the moment. Please, consider using the Expeditor integrator’s message splitting option to transfer larger files (see chapter Transfer of large files in multiple messages)
Memory usage and queue size
If a local micro broker queue in Expeditor integrator contains a large number of messages with large payloads, e.g. around 4000 messages of size 10K each and the JVM HeapSize is set to 256m, an OutOfMemory issue may occur while trying to "browse" the messages using the Expeditor integrator UI (Monitor tab | Live view). The same issue can occur if external tools like HermesJMS are used to browse the queues (http://www.hermesjms.com).
This OutOfMemory can be avoided to some extent by increasing the JVM HeapSize (see chapter 5.12.3 Running the Expeditor integrator with limited or increased memory). However, if the number of large messages continues to increase, the OutOfMemory error may happen during queue browsing even after increasing the HeapSize. Therefore, it is recommended to avoid browsing local queues under high memory usage conditions.
Graphical User Interface
The Lotus Expeditor integrator Web Application based GUI is accessed with Web Browser. The recommended browser is Mozilla Firefox v2 and v3 (from http://www.mozilla.com).
It is recommended for Microsoft® Internet Explorer (from http://www.microsoft.com) to change the browser settings to:
Tools | Internet Options | Temporary files | Settings | Check for newer version available | select Every visit to the page
Script execution must be allowed and cookies should be enabled in the browser for correct presentation of the GUI.
Expeditor integrator is based on Eclipse RCP. For some functions which are use by the Expeditor integrator GUI, language settings are retrieved from the Eclipse environment settings. Browser language settings are ignored in this case. If the used browser for accessing Expeditor integrator is set to language A but Expeditor integrator runtime (Eclipse RCP) uses language B the basic GUI layout will show up in language A and some called Expeditor functions will be displayed in language B.
It is therefore recommended that Administrators use the local browser on the Expeditor integrator instance or make sure that their browser is set to the same language as the accessed integrator instance.
Operating system service support under Windows 2008
Installation and operation of Expeditor integrator as Windows 2008 system service is only working under the Administrator account and in non-console mode (no interaction with desktop, no OSGi console is displayed).
The console mode can only be used through the Windows
Interactive Service Detection service which needs to be enabled for this case (see chapter
2.3.4.1 Install Expeditor integrator as Interactive Windows Service). A remote access to the OSGi console can also be configured (see chapter
5.12.7 Remote Access to OSGi console when running as a Service).
Using FTP Adaper in Passive Mode (Windows 2008)
Expeditor integrator may not be able to maintain connections to your FTP server in passive mode with the standard Windows Firewall settings being active (e.g. 4690 FTP Server). You must enable the integrator eclipse executable to access required ports and connections. The FTP TCP connection to/from your FTP server should be enabled.
You can test this by using the Windows FTP client. Open a connection and try to list files in remote FTP directories in passive mode (e.g. ftp myftpserver.com | user/pwd | quote pasv | ls). If the ls command does not return anything and you receive the following error instead, the local FTP client is unable to access any files:
425 Can’t build data connection: error 0.
The same error is displayed by the Expeditor integrator FTPAdapter when it can not access the remote FTP file:
…WARNING CWPIR006W: Activity xxx_FileWriteToFtp_xxxUpdate failed to write the file using the adapter, attempt x of y, reason = com.ibm.rcp.integrator.resourceadapters.ResourceException: CWPIR15576E: FTP adapter could not write the file mytargetftpdir/mytargetftpfile.txt to the FTP server, reason = 425 Can’t build data connection: error 0…
Talk to your administrator and enable this connection for Expeditor integrator in the local Firewall on Windows 2008 Server, e.g.
Inbound Rules: Allow program
/rcp/eclipse/plugins/com.ibm.rcp.base_./win32/x86/exlipse.exe
opening connections (in Public Profile)
Lotus Expeditor micro broker Acknowledgement Timeouts
Exception: Negative Acknowledgement for micro broker client with reason code 5
The MQTT_RETRY property specifies for how long the micro broker client should wait for the back-end Message Broker to acknowledge requests (in milliseconds). The default value is set to 15000 milliseconds.
If the client does not receive an Acknowledgement from the Message Broker within this default interval the client will throw an exception. This might be caused due to high load and limited resources on the Message Broker.
If these cases can be foreseen, it is recommended to configure a higher acknowledge waiting interval.
The Expeditor integrator configuration provides the MicrobrokerRetryInterval parameter (in milliseconds) for the Messaging Service that allows adjusting this to a higher value, e.g.
MicroBrokerRetryInterval=60000
Blocking dynamic queue creation
Administrators can disallow creation of dymanic micro broker queues by JMS/ MQTT clients (see parameter in chapter 4.3.1). In such cases, the client will get a JMSException stating:
Create consumer/producer failed: Not authorized.
If the client is a buffered sender, it will store the messages in a local buffer. Only when it tries to create a producer at the broker (Expeditor integrator) for a dynamic queue, the broker throws a MqttNegativeAcnowledgementException with ReasonCode=6 which is printed on the client console. An error message is also logged in the JMS trace file i.e. Clients are not authorized to create dynamic queues.
The user should note that despite this problem during the creation of a dynamic queue at the broker, there will be no error in the console when the client is storing the messages in the buffer before sending. As a result, the messages will remain in the client’s local buffer and will not be sent to the broker queue.
First Aid Guide
This is a little summary of steps that should be considered before contacting support.
Where to look first in case of Problems
Logging information is provided on component level as well as through the Custom Log Service and the Expeditor integrator Tivoli Monitor Service (iTMS). Basically, all application bundles write their log information to the standard OSGi Log Service. This logging information is kept in the Eclipse workspace folder (see folder structure in chapter Expeditor integrator file and directory structure). Furthermore, IBM Lotus Expeditor components do write log messages to independent log files. Please, refer to the following table for an overview of the available log files.
Table 51: Log information of the Expeditor integrator
|
Component and Description
|
Log information (log file path / name) |
Comment
(e.g. size restricted by property)
|
|
OSGi Log Service
(standard Eclipse and OSGi logging)
|
Information is logged in Eclipse log under <XPDINTEG_HOME>/workspace/logs |
Directory for log information from standard Eclipse and OSGi Core (OSGi Log Service)
The following properties can be edited in <XPDINTEG_HOME>/rcp/eclipse/plugins/com.ibm.rcp.base_<version><build>/config/pbuilder/ rcpinstall.properties
Expeditor integrator default:
com.ibm.rcp.core.internal.logger.boot.RCPLogHandler.size=4000000
com.ibm.rcp.core.internal.logger.boot.RCPLogHandler.count=6
com.ibm.rcp.core.internal.logger.boot.RCPTraceHandler.size=4000000
com.ibm.rcp.core.internal.logger.boot.RCPTraceHandler.count=6
The above (default) setting generates a maximum of 6 log and trace files with a maximum size of 4MB for each file (circular logging/tracing). Changes of size and number of log and trace files can be adjusted with these parameters.
|
|
Expeditor integrator startup problems
|
<XPDINTEG_HOME>/workspace/logs/StartupProblems.log |
e.g. due to config file parsing errors
Log file of max. 4kB size (wraps around when limit is reached)
|
| Custom Log Service |
Log queue name is configured in XPDinteg.xml with parameters
MqEventQName and MqLogQJndi
|
If bridged to messaging back-end, this may also be written to back-end XPDINTEG.LogQ or TEC console events.
Logging is configured in the XPDinteg.xml file
MicrobrokerQueueDepth=40000, LogThreshold=1 (Error), MonitoringBundles, MonitoringStatus, MonitoringLogTopicList
|
| TEC Console
(if the integrator Tivoli Mon Svc is used)
|
consult the TEC and look for XPDINTEG_LOGEVENT and XPDINTEG_BUNDLEEVENT events |
(parameters of Log Service apply) |
| TEC cache file
(stores events on disk when they cannot be sent to the event server)
|
<XPDINTEG_HOME>/persistent/tec/tivoli.send.cache |
Maximum size is 64KB for the Expeditor integrator v1.0.
If this is a requirement for future releases, a parameter could be provided that can be set in the XPDinteg.xml file.
|
| TEC log and trace files |
<XPDINTEG_HOME>/persistent/tec/eif.log and
<XPDINTEG_HOME>/persistent/tec/eif.trc
|
TEC libraries do not provide options to limit the trace and log file size.
The Expeditor integrator v1.0 passes the log file names as parameters on to the used TEC libraries. Currently, there is no way to limit the size of these two files (even from XPDinteg).
It is recommended to turn the logging and tracing only on in defined environments by setting TecLogLevel=ALL and TecTraceLevel=ALL (in XPDinteg.xml) and OFF again under production conditions.
|
| Microbroker and –bridge
(log contain information about the broker, connection to the back-end and message handling)
|
Log file under <XPDINTEG_HOME>\persistent\microbroker\XPDinteg_broker
Diagnostic information under
<XPDINTEG_HOME>/persistent/microbroker/XPDinteg_broker/diagnostics
|
-Location set with <persistence-directory>
-Size set with MicrobrokerLogSize in XPDinteg.xml
|
Transaction Service logs
(logs information about the status of transactions, errors and warnings)
|
Log files under <XPDINTEG_HOME>/persistent/txlogdir\partnerlog
and <XPDINTEG_HOME>/persistent/txlogdir/tranlog
|
Set in in XPDinteg.xml:
-TransactionLogDirectory=location of log files
-TransactionLogSize=file size limit in KB
|
The used log file language can be enforced by adding the osgi.nl parameter, e.g.
osgi.nl=en_US
in the /rcp/eclipse/plugins/com.ibm.rcp.base./config.ini file.
Using the provided Maintenance tool for collecting log and trace data
The <XPDINTEG_HOME>/XPDintegMaint.bat is provided as a part of the com.ibm.rcp.integrator.tools-feature. This can be used during maintenance for
- backing up and deleting log files
- ...deleting log files
- ...turning on/off MQJMS trace
The following files must exist as prerequisite:
- <XPDINTEG_HOME>/XPDintegMaint.bat
- <XPDINTEG_HOME>/maintenance/mqjmsOFF.txt
- <XPDINTEG_HOME>/maintenance/mqjmsON.txt
- XPDintegTools.jar by extracting the Expeditor integrator tools
- (<XPDINTEG_HOME>/tools/XPDintegTools.zip)
- <XPDINTEG_HOME>/workspace folder (This folder only exists after Expeditor integrator was started at least ones after platform reset.).
Steps for using XPDintegMaint.bat:
(All XPDintegMaint.bat options should be used when Expeditor integrator is not running.)
1. Backup and delete the following log files:
- <XPDINTEG_HOME>/workspace/logs
- <XPDINTEG_HOME>/FFDC
- <XPDINTEG_HOME>/mqjms.log
These files are deleted together with
*.lck and trace files, e.g. <XPDINTEG_HOME>/trace.xml, <XPDINTEG_HOME>/trace.xml.1 and <XPDINTEG_HOME>/ffdc.xml. These trace files are not required for the maintenance logging.
2. Execute the following command from <XPDINTEG_HOME> to create and backup log files:
XPDintegMaint.bat -backupAndDeleteLogs <LOG_BACKUP_DIR>
3. Delete log files without creating back up. Run:
XPDintegMaint.bat –deleteLogs
..to delete the following files:
- <XPDINTEG_HOME>/workspace/logs
- <XPDINTEG_HOME>/FFDC
- <XPDINTEG_HOME>/mqjms.log
- <XPDINTEG_HOME>/trace.xml
- <XPDINTEG_HOME>/trace.xml.1
- <XPDINTEG_HOME>/ffdc.xml
Note: This option should be used with care because deleting any logs may cause losing valuable information for the support teams.
Turning MQJMS Trace On/Off in production environment
If messaging connection problems continue to exist, detailed logging and tracing can be used for further investigation (see Table_51)
The recommended method in production environments for turning MQJMS Tracing on or off without resetting the Expeditor integrator runtime is to manually update the <XPDINTEG_HOME>/workspace/.config/rcpinstall.properties file.
Follow these steps:
1. Turning on MQJMS trace:
- Stop Expeditor integrator
- Add the following two lines to the <XPDINTEG_HOME>/workspace/.config/rcpinstall.properties file:
-DMQJMS_TRACE_LEVEL=base
-DMQJMS_TRACE_DIR=<TRACE_DIR>
- for e.g., add these lines for setting “C:/temp” as the <TRACE_DIR>:
-DMQJMS_TRACE_LEVEL=base
-DMQJMS_TRACE_DIR=C:/temp
- Start Expeditor integrator
- Start the micro broker trace tool:
- If this is the first time you have enabled tracing, you may need to unzip the XPDintegTools.zip file contained within the tools directory (<XPDINTEG_HOME>/tools).
- Additionally, configure the MBTrace.properties file prior to launching the batch file. Execute the MBtrace.bat (or MBTrace.sh on Linux). Details are provided in chapter Using the micro broker trace utility (MBtrace).
2. Turning off MQJMS trace:
- Stop Expeditor integrator
- Remove the following two lines above from <XPDINTEG_HOME>/workspace/.config/rcpinstall.properties file
- Re-start Expeditor integrator
3. After problem occurrence, execute the following commands in the OSGi console:
The dump threads command should be executed two more times. Ideally, execute the command thirty seconds after the previous command.
Enable Expeditor integrator tracing
Detailed tracing for Expeditor integrator can be switched on by adding the following line to
/workspace/.config/rcpinstall.properties
file:
com.ibm.rcp.integrator.level=FINEST
This enables tracing on FINEST level after Expeditor integrator is restarted.
If the trace level setting is supposed to be a permanent setting which also survives a platform RESET, the above statement must also be added to
\rcp\eclipse\plugins\com.ibm.rcp.base_\rcpinstall.properties
In this case, an Expeditor integrator platform RESET enables it (stop Expeditor integrator, run /resetscript/XPDintegReset.bat/.sh and start integrator again.).
In order to include the Java source code line number in displayed Java exceptions, the following parameter must be added to the /rcp/eclipse/plugins/com.ibm.rcp.j2se../jvm.properties file:
vmarg.Xnolinenumbers=-xnolinenumbers
Enable micro broker tracing
There is also tracing for the included IBM micro broker available. This can also be enabled by editing the
/workspace/.config/rcpinstall.properties
file. Please add the following line to enable FINEST tracing:
com.ibm.micro.level=FINEST
Now, the Expeditor integrator runtime needs to be restarted to activate this setting.
If the trace level setting is supposed to be a permanent setting which also survives a platform RESET, the above statement must also be added to
\rcp\eclipse\plugins\com.ibm.rcp.base_\rcpinstall.properties
In this case, an Expeditor integrator platform RESET enables it (stop Expeditor integrator, run /resetscript/XPDintegReset.bat/.sh and start integrator again.).
For more detailed analyses, an additional trool is shipped with Expeditor integrator that allows generation of very detailed debug and trace information for micro broker (see MBtrace tool in APPENDIX B – Technical Details, chapter Using the micro broker trace utility (MBtrace)).
Enable MQJMSTRACE
It’s also possible to enable the MQJMSTRACE which will create detailed trace files in the specified directory. In order to do so, follow the steps below.
Create a folder for storing the trace file (e.g. “C:/mqjmstrace”), referenced here as .
Create a trace configuration file (e.g. “trace.config”), referenced here as , and add the following lines:
com.ibm.msg.client.commonservices.trace.outputName=
com.ibm.msg.client.commonservices.trace.limit=52428800
com.ibm.msg.client.commonservices.trace.count=10
com.ibm.msg.client.commonservices.trace.standalone=TRUE
com.ibm.msg.client.commonservices.trace.level=TRACE_ALL
com.ibm.msg.client.commonservices.trace.status=ON
Now edit the file /workspace/.config/rcpinstall.properties or in order to make this setting permanent, edit the file \rcp\eclipse\plugins\com.ibm.rcp.base_\rcpinstall.properties. Add the following line:
-Dcom.msg.client.config.location=file:/
(e.g. -Dcom.msg.client.config.location=file:/C:/mqjmstrace/traceconfig)
After restarting, respectively resetting the platform, the trace files will be created in the configured directory.
For further information refer to http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/index.jsp?topic=/com.ibm.mq.csqzaw.doc/jm10770_.htm
Clean up folders, messages and restart the Expeditor integrator
The following steps are required to manually reset the Expeditor integrator to its initial state:
- Delete content of <XPDINTEG_HOME>/workspace directory
- Delete content of <XPDINTEG_HOME>/persistent directory
- use a working version of the config\XPDinteg.xml file
- Run resetscript/XPDintegReset.bat (which includes the first two steps if the default directories are used)
- Start Expeditor integrator by either starting the service/daemon or manually running the XPDintegStart.bat/sh.
Please, refer to the installation and set-up description in the Installing and Configuring the IBM Lotus Expeditor integrator platform (see section Installing Expeditor integrator).
Enable Expeditor Webcontainer tracing
IBM Lotus Expeditor integrator contains the Webcontainer as runtime environment for Servlets. It also provides tracing facilities which can be enabled by setting the level to FINEST in rcpinstall.properties: by adding this line:
com.ibm.ejs.level=FINEST
in /rcp/eclipse/plugincom.ibm.rcp.base_/rcpinstall.properties (Activation of setting requires Expeditor integrator RESET).
or
in /workspace/.config/rcpinstall.properties (Setting is active after restart of Expeditor integrator, but would be overwritten by Expeditor integrator RESET).
Line numbers can be enabled in the stack trace by commenting out two properties in file /rcp/eclipse/plugins/com.ibm.rcp.j2se.linux.x86_/jvm.properties:
#vmarg.Xnolinenumbers=-Xnolinenumbers
#vmarg.Dshare=-Xshareclasses:name=xpdplat_.jvm,controlDir=${prop.jvm.shareclasses.loc},groupAccess,keep,singleJVM,nonfatal
(see https://www.ibm.com/developerworks/mydeveloperworks/blogs/dpi/entry/expeditor_and_notes_stack_traces_unknown_source3?lang=en )
Obtain an ISA Collection using the OSGi Console
The IBM Service Assistance (ISA) is provided as plug-in and can be used directly in the OSGi console of Lotus Expeditor integrator (see section Expeditor Integrator console commands). Follow these steps to collect maintenance data for the IBM Support:
1. If not yet started, start IBM Lotus Expeditor integrator. If Expeditor integrator was started by the operating system service in non-Console mode, stop Expeditor integrator, switch to console mode and start it again (see chapter
Starting, Stopping, Resetting Expeditor integrator).
2. Run the following commands in the OSGi console:
start com.ibm.esupport.client.product.SSC4TNF
collectisa
Figure 41: Start IBM Support Assistent in OSGi console
3. When the ISA collection completes, a message will be displayed with the location of the ISA collection, e.g.
"The ISA Collection is complete. The output is located at C:\IBM\Lotus\Integrator\workspace\autopd\collector_port61_2010.09.07-11.13.52.120.zip"
4. Now, provide the ISA Data (zip fle) to the IBM Support by uploading the zip file using the IBM SR tool or e-mail the file to lotus_support@ecurep.ibm.com. If sent via e-mail, make sure that the PMR number exists in the subject field.
Gathering data for Expeditor Integrator is hanging
If the runtime hangs repeatedly collecting the JVM dump is very helpful for analyzis. If this is not done by default, triggering the dump and core file creation might be configured in the JVM properties by adding the line “-XDump:system+java:events=user” to the jvm.properties file. This will generate the system core and java dump files on user signals ([Ctrl]+[C] or kill -3).
On Windows this file is located under “\rcp\eclipse\plugins\com.ibm.rcp.j2se.win32.x86_1.6.0.20120806a-\jvm.properties”. For further setup information refer to http://www-01.ibm.com/support/docview.wss?uid=swg21222470. For further information about the relevant data, refer to http://www-01.ibm.com/support/docview.wss?uid=swg21222656.
On Linux this file is located under “/rcp/eclipse/plugins/com.ibm.rcp.j2se.linux.x86_1.6.0.20120806a-/jvm.properties”. For further setup information refer to http://www-01.ibm.com/support/docview.wss?uid=swg21222462. For further information about the relevant data, refer to http://www-01.ibm.com/support/docview.wss?uid=swg21222466
Gathering data for Expeditor Integrator is crashing
If the runtime is crashing repeatedly, the generation of the system core, the java dump and a snapshot of the memory might be triggered with the runtime crash for further analyzis. This is configured by adding the following line to the jvm.properites file “-XDump:java+system+snap:events=user+gpf+abort”
On Windows this file is located under “\rcp\eclipse\plugins\com.ibm.rcp.j2se.win32.x86_1.6.0.20120806a-\jvm.properties”. For further setup information refer to http://www-01.ibm.com/support/docview.wss?uid=swg21222468. For further information about the relevant data, refer to http://www-01.ibm.com/support/docview.wss?uid=swg21222654.
On Linux this file is located under “/rcp/eclipse/plugins/com.ibm.rcp.j2se.linux.x86_1.6.0.20120806a-/jvm.properties”. For further setup information refer to http://www-01.ibm.com/support/docview.wss?uid=swg21222461. For further information about the relevant data, refer to http://www-01.ibm.com/support/docview.wss?uid=swg21222464.
Uninstallation Issues
If IBM Lotus Expeditor integrator un-installation procedure does NOT remove all files, it is highly recommended that Expeditor integrator administrators/users manually remove the files left on the file system after the un-installation has been completed.